This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @VincentG1234.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

It will be great to get an example of "How to get the JSONL" because i don't find solutions in litellm for example.

Yeah that’s true, most frameworks won’t produce this exact JSONL directly.

That’s kind of intentional. The idea here is to define a minimal, framework-agnostic canonical replay format, not something tied to a specific tracing stack.

In practice, the required fields already exist almost everywhere (timestamp, input token count, output token count), just under slightly different names, so a small mapping step is usually enough.

I agree it’s not the best UX on its own, but it felt like the right minimal base for the feature. Then we can iterate on top of it with helpers / converters for common sources like LiteLLM or Langfuse. And we can extend it later (e.g. optional prompt field, multiple timestamp formats, richer metadata) without breaking the core idea.

But happy to adjust the direction if maintainers prefer something more opinionated or integrated from the start.

Thanks a lot for the detailed review, I really appreciate your time.

I’m fully aligned with your feedback, especially on the replay handling in the entrypoint, which is a key part of the PR. I agree that introducing a special case here is not ideal and should be avoided.

I’ll refactor this to make it cleaner and better aligned with the existing design.

Hi @sjmonson,

Thanks again for the feedback.

I’ve completed the refactor and addressed the main review points.

Key updates:

• replay now uses the standard request loader flow (removed special-case path)
• data_samples handles dataset truncation, max_requests stays runtime-only
• shared trace loading moved to guidellm.utils.trace_io
• simplified replay constraints
• tests/docs cleaned up accordingly

Would appreciate another look when you have time.

Optional: I also put together a small Colab notebook to try the feature quickly if useful:
https://colab.research.google.com/drive/1hOY9Kg5BVYz4BZzJLrcUKWiMDR2_lU7V?usp=sharing

I tried running a test with this dataset and got a deadlock at the start of the benchmark. Here is a jsonl version for testing: data.jsonl.gz.

I tried running a test with this dataset and got a deadlock at the start of the benchmark. Here is a jsonl version for testing: data.jsonl.gz.

Thanks for testing this and for sharing the dataset.

I can reproduce the issue on my side as well, including with a smaller subset of the trace. I’ll investigate it and work on a fix as soon as possible.

At first glance, this seems related to how replay handles large/bursty traces and high-token-count requests.

I’ll follow up once I have a clearer diagnosis and a fix.

Hi @sjmonson
This commit fixes two issues found while replaying the shared JSONL trace.

First, synthetic prompt generation now builds one reusable base prompt and creates each request prompt by adding a unique prefix before slicing to the requested input length. This keeps prompts cache-resistant while avoiding the previous expensive per-request generation path.

Second, trace replay is temporarily limited to one process. With multiple processes, there is currently a race condition where some scheduled requests can be consumed out of order or never sent, which leaves the benchmark waiting forever. Capping replay to one process is a workaround, but it makes the benchmark complete reliably; the only expected limitation is for extreme traces where one scheduling process may become a bottleneck.

I tested this with the shared JSONL trace: the benchmark no longer hangs and starts correctly after roughly 50 seconds. On a representative subset, the previous prompt generation path was at least 10x slower...

First, synthetic prompt generation now builds one reusable base prompt and creates each request prompt by adding a unique prefix before slicing to the requested input length. This keeps prompts cache-resistant while avoiding the previous expensive per-request generation path.

Yeah... I don't love this idea, but fine for now. I see the dataset generation as temporary/fallback anyways since what we want longer term is to base the tokens off of the mooncake token ids.

Second, trace replay is temporarily limited to one process. With multiple processes, there is currently a race condition where some scheduled requests can be consumed out of order or never sent, which leaves the benchmark waiting forever. Capping replay to one process is a workaround, but it makes the benchmark complete reliably; the only expected limitation is for extreme traces where one scheduling process may become a bottleneck.

Also works for now. Fixing this requires a way for the dataset to inform on request scheduling so we'll scope something out.

For multiprocessing, I think I may have found a fairly minimal approach that avoids the replay deadlock while keeping the implementation relatively clean, but I agree it’s probably better scoped for a follow-up PR.

For the Mooncake token-id direction, unless I miss something, I think the current prefix invalidation approach can remain compatible with a more structure-aware strategy later on.

Roughly, unrelated prompts would still receive different invalidating prefixes, while prompts sharing a common prefix could intentionally reuse the same initial invalidating block and only diverge later with unique suffix markers.

Example:

base prompt:
[A, B, C, D, E]

prompt 1:
[x, A, B, C]

prompt 2 (totally unrelated to prompt 1):
[y, A, B, C, D]

prompt 3 (shares the same prefix structure as prompt 2 up to block D):
[y, A, B, C, z, D, E]

This keeps prompts cache-resistant globally while still allowing controlled shared-prefix behavior between related requests. The same idea could likely be extended recursively for deeper shared-prefix structures.

augment review

^ Trialling some new AI review tools. I resolved the comments that I believe don't need to be fixed but check the remaining one.

base prompt: [A, B, C, D, E]

prompt 1: [x, A, B, C]

prompt 2 (totally unrelated to prompt 1): [y, A, B, C, D]

prompt 3 (shares the same prefix structure as prompt 2 up to block D): [y, A, B, C, z, D, E]

There are two main problems with this strategy:

When you prepend tokens to a sequence there is no guarantee that you will end up with the same stream of tokens once you detokenize -> retokenize. Your prefix can merge with the first token of the prompt to form a different token which will cascade though the rest of the prompt. Usually this just results slight token count inaccuracies though occasionally you get a wider gap. In the past versions of GuideLLM we did something similar and could see as high as 25 tokens difference between requests token count and actual token count on a very long prompt. The severity of this issue depends on the tokenizer; many tokenizers are mostly unaffected as long as there is a space between the prefix and the prompt, others have many tokens that cross word boundaries.

The other issue is that by using a single static block the bytes to tokens (and words to token) ratio is static; which can matter for all of the code around the LLM (templating, HTTP pathways, etc). Its not realistic for this ratio to be static so the test could produce biased results.

I don't think these are deal-breakers to getting this code in but it is something we will probably address down the line.

I fixed the small issue around the trace source

The other issue is that by using a single static block the bytes to tokens (and words to token) ratio is static; which can matter for all of the code around the LLM (templating, HTTP pathways, etc). Its not realistic for this ratio to be static so the test could produce biased results.

Thanks, this is really interesting, I had never really considered the bytes/token ratio aspect and its impact. If needed here, using a small reusable pool of diverse base prompts instead of a single static one could already help.